Skip to content

chore(airflow): Extend providers/extras for 3.0.6 #1336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Nov 11, 2025
Merged

chore(airflow): Extend providers/extras for 3.0.6 #1336

merged 15 commits into from
Nov 11, 2025
+67 −6

Conversation

@adwk67
Copy link
Member

@adwk67 adwk67 commented Nov 6, 2025 •
edited
Loading

Description

This PR extends the list of extras/providers that will be installed when building the image for Airflow 3.0.6.
This is the only version that has been changed as

  • Airflow 2.x versions are deprecated and will not be supported as of SDP 26.3
  • Airflow 3.0.1 is marked as experimental and 3.0.6 should be preferred over it

Fixes stackabletech/airflow-operator#705

🟢 Local tests (all 3.0.6 ones):

--- PASS: kuttl (3451.47s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-3.0.6_openshift-false (151.02s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-server-verification-tls_openshift-false_executor-celery (344.73s)
        --- PASS: kuttl/harness/opa_airflow-3.0.6_opa-latest-1.8.0_openshift-false (236.56s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-no-tls_openshift-false_executor-kubernetes (247.23s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-no-tls_openshift-false_executor-celery (217.06s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-insecure-tls_openshift-false_executor-kubernetes (203.87s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-insecure-tls_openshift-false_executor-celery (199.12s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-3.0.6_openshift-false (296.97s)
        --- PASS: kuttl/harness/overrides_airflow-latest-3.0.6_openshift-false (332.07s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-3.0.6_openshift-false_executor-kubernetes (175.69s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-3.0.6_openshift-false_executor-celery (173.23s)
        --- PASS: kuttl/harness/remote-logging_airflow-latest-3.0.6_openshift-false_executor-kubernetes (253.39s)
        --- PASS: kuttl/harness/remote-logging_airflow-latest-3.0.6_openshift-false_executor-celery (282.43s)
        --- PASS: kuttl/harness/triggerer_airflow-latest-3.0.6_openshift-false_executor-kubernetes (198.79s)
        --- PASS: kuttl/harness/triggerer_airflow-latest-3.0.6_openshift-false_executor-celery (194.84s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.0.6_openshift-false_executor-celery (540.37s)
        --- PASS: kuttl/harness/smoke_airflow-3.0.6_openshift-false_executor-kubernetes (207.59s)
        --- PASS: kuttl/harness/smoke_airflow-3.0.6_openshift-false_executor-celery (294.79s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-3.0.6_openshift-false_executor-kubernetes (292.83s)
        --- PASS: kuttl/harness/resources_airflow-latest-3.0.6_openshift-false (180.65s)
        --- PASS: kuttl/harness/oidc_airflow-latest-3.0.6_openshift-false (277.30s)
        --- PASS: kuttl/harness/logging_airflow-3.0.6_openshift-false_executor-kubernetes (505.75s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.6_openshift-false_executor-celery (212.47s)
        --- PASS: kuttl/harness/logging_airflow-3.0.6_openshift-false_executor-celery (390.04s)
        --- PASS: kuttl/harness/ldap_airflow-latest-3.0.6_ldap-authentication-server-verification-tls_openshift-false_executor-kubernetes (157.87s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.6_openshift-false_executor-kubernetes (213.32s)
PASS

Connections are now available in the UI:
image

Definition of Done Checklist

Note

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.

Please make sure all these things are done and tick the boxes

  • Changes are OpenShift compatible
  • All added packages (via microdnf or otherwise) have a comment on why they are added
  • Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
  • All packages should have (if available) signatures/hashes verified
  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

boil build <IMAGE> --image-version <RELEASE_VERSION> --strip-architecture --load
kind load docker-image <MANIFEST_URI> --name=<name-of-your-test-cluster>

See the output of boil to retrieve the image manifest URI for <MANIFEST_URI>.

@adwk67 adwk67 changed the title extension to list for 3.0.6 chore(airflow): Extend providers/extras for 3.0.6 Nov 6, 2025
@adwk67 adwk67 marked this pull request as ready for review November 6, 2025 16:45
@adwk67 adwk67 self-assigned this Nov 10, 2025
@adwk67 adwk67 moved this to Development: Waiting for Review in Stackable Engineering Nov 10, 2025
@maltesander maltesander self-requested a review November 10, 2025 10:50
@maltesander maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Nov 10, 2025
maltesander
maltesander requested changes Nov 10, 2025
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and works. This is very cumbersome to check and maintain.

Can we split the AIRFLOW_EXTRAS parameters to be closer to the actual docs like:

  • AIRFLOW_EXTRA_CORE
  • AIRFLOW_EXTRA_META
  • AIRFLOW_EXTRA_PROVIDER_APACHE
  • AIRFLOW_EXTRA_PROVIDER_EXTERNAL_SERVICES
  • AIRFLOW_EXTRA_LOCALLY_INSTALLED_SOFTWARE
  • AIRFLOW_EXTRA_OTHER

I think this would already help to make it easier. This could be concatenated in the docker image into AIRFLOW_EXTRA.

@adwk67
Copy link
Member Author

adwk67 commented Nov 10, 2025 •
edited
Loading

Tested and works. This is very cumbersome to check and maintain.

Can we split the AIRFLOW_EXTRAS parameters to be closer to the actual docs like:

  • AIRFLOW_EXTRA_CORE
  • AIRFLOW_EXTRA_META
  • AIRFLOW_EXTRA_PROVIDER_APACHE
  • AIRFLOW_EXTRA_PROVIDER_EXTERNAL_SERVICES
  • AIRFLOW_EXTRA_LOCALLY_INSTALLED_SOFTWARE
  • AIRFLOW_EXTRA_OTHER

I think this would already help to make it easier. This could be concatenated in the docker image into AIRFLOW_EXTRA.

Done in 4b1d932

Built 3.0.6 and 3.0.1 and tested with smoke tests:

--- PASS: kuttl (443.57s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.1_openshift-false_executor-celery (232.51s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.6_openshift-false_executor-celery (235.56s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.6_openshift-false_executor-kubernetes (201.50s)
        --- PASS: kuttl/harness/external-access_airflow-3.0.1_openshift-false_executor-kubernetes (206.91s)
PASS

I missed one provider - tabular - which has been added :)

@adwk67 adwk67 requested a review from maltesander November 10, 2025 16:20
maltesander
maltesander reviewed Nov 11, 2025
View reviewed changes
maltesander
maltesander approved these changes Nov 11, 2025
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@adwk67 adwk67 enabled auto-merge November 11, 2025 08:53
@adwk67 adwk67 added this pull request to the merge queue Nov 11, 2025
Merged via the queue into main with commit 0fc5dc0 Nov 11, 2025
3 checks passed
@adwk67 adwk67 deleted the feat/airflow-add-providers branch November 11, 2025 08:54
@adwk67 adwk67 moved this from Development: In Review to Development: Done in Stackable Engineering Nov 11, 2025
@lfrancke lfrancke moved this from Development: Done to Acceptance: In Progress in Stackable Engineering Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maltesander maltesander maltesander approved these changes

Assignees

@adwk67 adwk67

Labels

None yet

Projects

Stackable Engineering
Status: Acceptance: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update Airflow providers

3 participants

@adwk67 @maltesander